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TECHNIQUES FOR PHOTOG RAM METRIC SYSTEMS 



CROSS-REFERENCE TO RELATED APPLICATIONS 

[001] The application claims priority of US provisional application 60/233471, filed September 
19, 2000 and entitled "A New General Purpose Photogrammetric 3D Measure System. 

FIELD OF THE INVENTION 

[002] The present invention relates to the measurement of any visible objects by 
measurement of points or similar structures. More particularly it relates to a method and 
apparatus for the measurement of 3D coordinates of points by analyzing images of 
same. 



can be accomplished with coordinate measuring machines (CMMs), teodolites, 



H photogrammetry, laser triangulation methods, interferometry, and other contact and 
H non-contact measurements. However all tend to be complex and expensive to 
f " implement in an industrial setting. 

[004] US patent: "4431290: Portable photogrammetry system", issued February 14, 1984, 
provides a short-range hand-held photogrammetry system consisting of two cameras in 
rigidly fixed positions at the remote ends if a hollow bar. The system provides for 
simultaneous image capture. 

[005] A number of useful methods also exist as exemplified by US patent: "4802757: System 
for determining the attitude of an airborne platform or the alike", issued February 7, 
1989. That patent describes a system that employs digital image grayscale correlation 
in order to relatively position the moving image sensor at the times of the successive 




3D Measurements are well known in the art and are widely used in the industry. The 
purpose is establishing the 3 coordinates of any desired point with respect to a 
reference point and coordinate system. As known in the prior art, these measurements 
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image captures. 

[006] However, applications of these systems and methods tend to be limited. Some are 
physically too large to be moved and easily applied, others require a lot of human 
intervention. Most require a relatively long data acquisition time where an object has to 
stand still. Furthermore, they are optimized for a specific object size. Thus, what is 
needed is a flexible, easily implemented system that can measure in a wide variety of 
industrial settings. A system performing measurements at sites radically varying by size 
and complexity, for example measurements in the construction industry as well as in 
continuous manufacturing processes, is needed. 



if! 



SUMMARY OF THE INVENTION 

[OOXp Accordingly, an object of the present invention is to provide a widely applicable and 
efficient techniques for systems based on photogrammetric principles, which can 
H! measure three-dimensional coordinates. 
[00811 It >s another object of the present invention for the techniques to apply to either 
m stationary or hand-held cameras. The techniques can use different means of image 
acquisition, such as photo camera, video camera, and x-ray, and the described 
methods are equally applicable to all modalities of image capture. 
[009J; According to a first broad aspect of the present invention, there is provided a method for 
0 performing camera position approximation. A method is taught for finding the position 
of an object in front of a camera. The starting point is an image taken by the camera in 
question for which the position is unknown. Three points are selected on the image and 
are projected into an image space by intersecting them with the known focal point from 
the image. The relationship between these three points is known. A first point, point A, 
is selected arbitrarily on a first of the three rays produced. Point A is preferably selected 
close to the image plane. A second point, point B, is found on a second of the three 
rays by respecting the known distance between two of the points selected on the image 
plane. In order to determine the location of point C, a circle is drawn, its axis on a path 
AB and its radius such that known distance AC and BC are respected. Point C is then 
identified as the point on the circle that is closest to the third ray, 
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[010] Preferably, the process is repeated by choosing different positions for point A in order to 
find the point C closest to the third ray. 

[011] According to a second broad aspect of the present invention, there is provided a 
method for reconstructing planar objects wherein the planar objects are captured in a 
plurality of two-dimensional images taken from different angles. The method comprises 
reconstructing a set of points in an image plane in three dimensions using standard 
photogrammetric techniques. The set of points should contain at least three points. A 
three-dimensional position for a selected point must then be calculated on the planar 
object by identifying the point on one of the two-dimensional images. The actual 
calculation comprises intersecting a ray that passes from a focal point of a camera 
through the selected point on the image plane and a plane that is calculated to fit well 
" with the set of points. 

la? 

[0#j According to a third broad aspect of the present invention, there is provided a method 

rii for precision estimation of a measured three-dimensional coordinate based on Monte- 

^ Carlo simulation. The first step is to choose a three-dimensional point in an imaged 

in 

I ' space and calculate the point's theoretical position in the image planes. This point 

s *\ becomes a simulation point. Reference points are then chosen in the image planes and 

H their positions are calculated. Tolerances for internal and external parameters of the 

sx 
i i £ 

n system (or camera) are then acquired from a system database. A suitable amount of 
error to positions and parameters are then added to the simulation, the error being 
determined according to the acquired tolerances. The simulation point is then 
reconstructed based on its position in images obtained through the simulation and a 
three-dimensional coordinate is produced. The last step is to compare the original 
three-dimensional point with the three-dimensional coordinate that was produced by the 
reconstruction to obtain a precision estimation. Repeating the precision estimation with 
different amounts of error and averaging out the results can produce a better estimation. 



BRIEF DESCRIPTION OF THE DRAWINGS 

[013] These and other features, aspects and advantages of the present invention will become 



-3- 



14222-2us 



SEP; 1 9. 2001 2:59PM 



SWABEY OGILVY MTL 514 288 8389 



NO. 0111 P. 10/20 



better understood with regard to the following description and accompanying drawings 



[014] FIG. 1 is schematic of the camera position approximation process; 
[01 5] FIG. 2 is a schematic of the reconstruction of planar objects process; 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[016] For the photogrammetric reconstruction of points in three dimensions, two or more 
photos of a scene are needed. Further, it is necessary to know the camera parameters 
and position for every photo taken. This requirement is in contradiction with the 
requirement for flexibility and mobility. Therefore, when camera positions are not known, 
camera position approximation is applied: along with the measured object, a sufficient 
$ number of reference points need to be present in the photos and they are used to find 
m the positions of mobile camera(s) after the images are taken. Note that reference points 
are only needed if the camera positions and parameters are not already precisely 
$ known. 



[017ji In general, calibration of the camera position and parameters is performed by using 
~ known reference points that make part of the scene. First, these points are identified in 
vli the photos. Their known positions in the scene (3D) and in the photos (2D) make it 
H possible to first find the approximation of the camera position for every photo. They are 
also used to adjust the internal camera parameters, which may vary from one photo to 
another, like the focus and principal point in the compensation phase. Further they are 
also used to adjust implementation specific parameters of the imaging device such as 
lens distortion correction parameters. The parameters are the radial, tangential and 
centering error correction parameters. 

[018] The Camera Position Approximation software determines the camera position from 
information about reference points in each of the images. The reference points can be 
identified in the image by pattern recognition methods and the system determines the 
camera position without any operator assistance. If the number of machine recognizable 



wherein: 
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reference points is not sufficient in one or more of the images then the user is required 
to identify the reference points. 

Figure 1 demonstrates the process of finding the position of an object in front of a 
camera. The starting point is an image taken by the camera in question for which the 
position is unknown. Three points are selected on the image and are projected into an 
image space by intersecting them with the known focal point from the image. The 
relationship between these three points is known. A first point, point A, is selected 
arbitrarily on a first of the three rays produced. Point A is preferably selected close to 
the image plane. A second point, point B, is found on a second of the three rays by 
respecting the known distance between two of the points selected on the image plane. 
In order to determine the location of point C, a circle is drawn, its axis on a path AB and 
its radius such that known distance AC and BC are respected. Point C is then identified 
as the point on the circle that is closest to the third ray. The process can be repeated 
by choosing different positions for point A in order to find the point C closest to the third 
ray. Please refer to Table 1 for the algorithm in pseudo-code. 

Program FlndCameraPosition(image, referenceElementModel): 

ReferencePointsArray = 

= correlation(image, referenceElementModel, reference_point_pattern) 

imagePointA = Select3ReferencePointsFromArray(ReferencePointsArray) 
imagePointB = Select3ReferencePointsFromArray(ReferencePointsArray) 
imagePointC = Select3ReferencePointsFromArray(ReferencePointsArray) 

rayA = new Ray(focus, imagePointA) 
rayB = new Ray(focus, imagePointB) 
rayC = new Ray(focus, imagePointC) 

for (all pointA on rayA) 
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sphereB = new Sphere (radius=d(A,B), center=pointA) 
pointB = cut_path_and_sphere_3D(rayB, SphereB) 
axisAB = new Path( through pointA and pointB) 

circleC = new Circle (radius=d(C,AxisAB), center=projection of C on axisAB) 
error = distance_path_to_circle(rayC, circleC) 
if (detect_local_minimum(error)) 

{ 

position = find_rot_transl(pointA, pointB, pointC) 
error_all = find_error_for_all_points(all points) 
if (detect_global_minimum(error_all)) 
store(position) 

} 



h Table 1 : Algorithm In Pseudo-Code 

[02(J|i Compensation is the process of calculating the internal camera parameters: the focal 
P length - F, and the principal point offsets - Xpp and Ypp. At the same time the estimated 
r camera positions are fine adjusted. During compensation parameters for all images are 
optimized in the same process and a common best fit is found. 

[021] There are two ways to calibrate a block of photos: either only by calibration of the block 
of photos (for camera parameters and positions) or by both pre-calibration of the 
camera and calibration of a block of photos. 

[022] Photo block calibration alone: Calibration of a block of photos is performed to obtain the 
internal camera parameters (Xpp, Ypp, F) and the external parameters (the positions 
from where the photos were taken - X.Y.Z.K.P.O). Reference points are identified in 
every photo. A data-fitting algorithm using the maximum likelihood method adjusts the 
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said parameters and positions in order to harmonize the mathematical model with the 
measured data. 

[023] PRE-Calibration and block calibration; Although the information in the block of photos is 
self sufficient for the calibration to be performed, sometimes it is advantageous to 
perform the Pre-calibration of the camera to improve further on the precision of the 
measurement. Pre-calibration of the camera parameters (Xpp, Ypp, F) can only be 
meaningful if the parameters are not changing. Auto-focus cameras, in general, cannot 
use pre-calibration. Pre-calibration is performed best with a large number of precise 
points that are well distributed in the field of view of the camera. A good set of reference 
points is the "calibration cube". The calculated camera parameters (Xpp, Ypp, F) are 
permanently stored in the camera properties. These parameters are reused every time 
jfj that the camera is used. They are considered as very precise. 

[024] The pre-calibrated parameters will enter the algorithm of the photo block calibration 
rij (camera parameters and position) with a very small standard deviation value, This 
means the camera parameters will strongly "stick" to the pre-calibrated values during 
a the calibration of a block. At the same time, predominantly the camera position will be 
"*f{ adjusted to fit the measured data best. 



[028) The "Internal Orientation Start Value" option leaves the choice to the user to choose 
[;! between the pre-calibrated ("Calibrated per camera") and the approximated value for 
the internal parameters. The pre-calibrated option will be the default if pre-calibration 
was performed for the active camera. If so, the proper parameter values that are stored 
in the project database will be applied. It should be used if the camera has been 
carefully pre-calibrated and remained untouched since. Otherwise, the "approximated" 
values for the internal parameters are to be used. 

[026] In a second embodiment the camera positions and parameters are fixed and known and 
no reference elements are needed in the images and the camera position 
approximation and compensation are not applied. 



-7- 



14222-2us 



SEP. 19. 2001 3:00PM SWABEY OGILVY MTL 5 14 288 8389 ^ NO. 0111 P. 14/20 




[027] In the reconstruction phase a point needs to be selected in at least two images taken 
from different angles in order to be restituted. Any triangulation-based algorithm can be 
applied to find the points' position in object space based on known camera position 
parameters and the point's position in multiple images. For point identification, images 
are displayed side by side. The identification of points in the multiple images can be 
performed in a number of modalities. In the manual modality the user is able to select 
any matching points on multiple images by clicking them with the mouse. 

[028] In the first automatic modality the system searches for points with a user specified 
image pattern in user specified areas of the images. Image patterns like points, crosses 
or circles can be specified as targets for the search algorithm. The system applies a 
pattern recognition method, like for example the normalized grayscale correlation. For 
increased precision subpixel methods are applied. These algorithms are precise up to a 
portion of a picture element - pixel. In the second automatic modality the user points out 
any object point in any of the images and the system identifies the same object point, if 
visible, in other images. In any of the modalities, the system allows the user to perform 
a manual correction of the point positions. Information about the relative orientation 
C) between the two images can be used to harmonize the orientation of the left and right 
P image by rotation. This is to facilitate the pattern recognition task. 

[029J: The algorithm used to reconstruct planar objects begins by reconstructing a set of 
points in an image plane in three dimensions using standard photogrammetric 
techniques. The set of points should contain at least three points. A three-dimensional 
position for a selected point must then be calculated on the planar object by identifying 
the point on one of the two-dimensional images. The actual calculation comprises 
intersecting a ray that passes from a focal point of a camera through the selected point 
on the image plane and a plane that is calculated to fit well with the set of points. 

[030] In another embodiment, the system performs a precision estimate of the reconstructed 
points. The system performs for each of the points of interest a simulation of the 
physical process of image capture, calibration and reconstruction. The simulation is 
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based on the given tolerances for the system parameters - e.g. reference points, 

[031] The method used for precision estimation of a measured three-dimensional coordinate 
is based on Monte-Carlo simulation. The first step is to choose a three-dimensional 
point in an imaged space and calculate the point's theoretical position in the image 
planes. This point becomes a simulation point. Reference points are then chosen in 
the image planes and their positions are calculated. Tolerances for internal and 
external parameters of the system (or camera) are then acquired from a system 
database. A suitable amount of error to positions and parameters are then added to the 
simulation, the error being determined according to the acquired tolerances. The 
simulation point is then reconstructed based on its position in images obtained through 
the simulation and a three-dimensional coordinate is produced. The last step is to 
compare the original three-dimensional point with the three-dimensional coordinate that 
was produced by the reconstruction to obtain a precision estimation. Repeating the 
precision estimation with different amounts of error and averaging out the results can 
H;f produce a better estimation. 

i, i 1 
"»ssr 

s : : 

[03|j In one variation of the invention, a structured light source is provided to project a 

"'=! multiplicity of target patterns. This is to facilitate the correlation of corresponding points 

h= in a multiplicity of captured images. 

[03§( As the reconstruction phase is done, the system allows the user to perform different 
tasks of data editing and analyzing such as: modification of the data origin and 
orientation, re-sampling, comparison with other model, export in different formats etc. 

[034] It will be understood that numerous modifications thereto will appear to those skilled in 
the art. Accordingly, the above description and accompanying drawings should be taken 
as illustrative of the invention and not in a limiting sense. It will further be understood 
that it is intended to cover any variations, uses, or adaptations of the invention following, 
in general, the principles of the invention and including such departures from the 
present disclosure as come within known or customary practice within the art to which 
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the invention pertains and as may be applied to the essential features herein before set 
forth, and as follows in the scope of the appended claims. 
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